1. Field of the Invention
This invention relates in general to the fields of Computer Operating Systems, Distributed Multi-processor Hardware Systems, Object Oriented Programming, and Virtual Memory Systems. In particular, the invention relates to improved techniques for establishing and efficiently extending file system functionality by structuring file systems as a set of dynamically configurable layers.
2. Background
The role of the operating system in a computer has traditionally been to efficiently manage the hardware resources (the central processing unit ("CPU"), memory and input/output devices). Since files are stored generally on a disk drive or magnetic tape system, this management function has included the role of managing the file system.
Object oriented operating systems, with microkernels which permit client level implementation of file systems, create complexities in memory management which clients have not had to deal with in the past. Moreover, on widely distributed computer networks, files resident on different computers produce unnecessary network communications traffic, without efficient local caching techniques. Nevertheless, micro-kernel systems provide clients with an environment which makes it possible to add functionality to existing file systems, to create new file systems and extend old file systems while minimizing the necessity to write new code to duplicate or significantly modify the existing file systems and existing caching capabilities.
This disclosure describes an extensible file system architecture and a method and apparatus for an extensible file system.
In reviewing the prior art a few definitions are in order. A file is a collection of related information defined by its creator. Commonly, files represent programs (both source and object forms) and data. Data files may be numeric, alphabetic, alphanumeric or binary; may be free-form or formatted rigidly. In general a file is a sequence of bits, bytes, lines, or records whose meaning is defined by the file's creator and user. A file is named and is referred to by its name. In most hierarchical file systems, the name of a file is the path name that describes how to locate the file in the system hierarchy. A path name is a sequence of component names separated by the "/" character. A full path name starts with a "/" character which specifies that a file is found by starting at the file system root and traversing the file tree, following the branches that lead to successive paths. For example, the path name "/usr/src/cmd/two.c" designates the "two.c". A file system then is an application program which knows how to manage a particular set of files, with their particular type of data coding and format and length and other particular characteristics.
In a UNIX.RTM. based system, the operating system interacts directly with the hardware providing common services to programs and isolating the software and users/clients from the hardware idiosyncrasies of a particular computer system. (UNIX is a registered trademark of UNIX System Laboratories, Inc.) In UNIX the operating system is commonly called the "system kernel" or just the "kernel", emphasizing its isolation from the user/client programs. The UNIX file system, located within the kernel, organizes the file structures. The file system has traditionally been responsible for managing the file system by: creating and deleting files and directories; providing support for primitive program routines for manipulating files and directories; mapping files onto disk storage; and general protection of the files by limiting access of programs, processes and users to the files. In modern operating systems, this file management function has included the use of a virtual memory subsystem.
Distributed computer systems, some with shared memory, and some with remotely accessible file systems, have led to the creation of "distributed file systems ("DFS")" to support the sharing of files by multiple users when the files are physically dispersed among the various computers of a distributed system. A DFS is a file system whose clients, servers and storage devices are dispersed among the machines of a distributed system. The location and multiplicity of the servers and storage devices is transparent to the client. For additional information on operating systems and file systems, see the text "Operating System Concepts" 3rd edition, by A.Silberschatz, J.Peterson and P.Glavin, 1991 Addison-Wesley Publishing Inc.
Typically, an operating system provides one or two types of file systems that are not modifiable by the user (extensible). In current systems such as UNIX, the file system provides a storage mechanism (in addition to a naming facility) that manages stable storage media and cooperates with the virtual memory system to cache data in memory. There are several architectures that allow for extending the functionality of the file system in one way or another. The Virtual File System (VFS) architecture was originally designed to accommodate multiple file systems within a UNIX kernel. That is, VFS could assist in adding an entire new file system, but it is extremely difficult to replace or enhance separate portions of the file system, as for example, keeping the physical disk management and installing a new directory layer. To add new file system functionality requires either modifying the existing file systems (requiring knowledge of current, multi-thousand line file systems and low level kernel debugging tools) or adding a new file system (again requiring the writing usually of multiple lines of code much of which duplicates existing file system support code).
Recently, a proposal was made to evolve VFS to support the implementation of new file systems in terms of pre-existing ones. Other systems such as the Apollo extensible IO system and the FICUS system allow for extending the file system in a limited manner. Such systems demonstrate along with other systems the need for extending the file system. However, the goals of these systems are necessarily limited. For additional information on FICUS and recent attempts to add functionality using a stackable filing layer framework see the following two papers: "A Layered Approach to File System Development" by John S. Heidemann and Gerald J. Popek, Department of Computer Science, University of California, Los Angeles, Technical Report CSD-910007, March 1991; and "File System Development with Stackable Layers", Draft report by John S. Heidemann and Gerald J. Popek, Department of Computer Science, University of California, Los Angeles, Mar. 28, 1992.
With the advent of micro-kernel operating systems, file systems are being implemented outside of the kernel in user level servers. These new file systems must solve a new set of problems to provide efficient performance. In addition however, the micro-kernel architecture provides the ability to add functionality to existing file systems to handle the necessary security or encryption requirements, as well as add new caching and coherency capabilities with minimal user or operating system disruption. Examples of new functionality that may need to be added to existing file systems include data or file compression, replication, encryption, distribution, and extended file attributes.
There are four broad requirements that are necessary for a flexible extensible file system architecture:
1. Leveraging existing file systems. One should be able to easily leverage existing file systems when introducing new file system functionality. This should be achievable without affecting the clients of the existing file systems.
2. Caching. For efficient performance reasons, the extensible file system architecture must define means for caching file data and attributes.
3. Coherency. Due to caching, distribution, and to multiple clients accessing the same data from different points of view, the architecture must define a framework for keeping file data and attributes coherent. However, the coherency policies should be left to the implementation of the file systems. Coherentin this case means that the data and attributes in all user's address spaces must be updated when any one user updates a field in their copy of the data or attributes.
4. Dynamic addition of functionality. It should be possible to add new functionality to a running system, and to dynamically extend the functionality of files. In addition, new file systems should be able to reside in the kernel or in user mode.